System performance in forms processing is typically measured through “inside out” or “white box” test scripting or costly 100% inspection scenarios. System level testing from ingest to population of the content management system is generally not performed, nor is statistical process control employed. This is primarily due to the lack of suitable test materials having a perfectly known result or “truth” with which to compare system performance results, particularly in sufficient quantities to allow statistically valid conclusions.
We have solved the above problems with this invention, which makes possible “outside in” or “black box” testing and cost-effective sampling for process quality assurance. This enables system level testing of all (or portions of) a forms data capture system and provides valid data for statistical process control. A preferred element of this invention is the incorporation of special engineered test materials such as a Digital Test Deck®, available from ADI, LLC of Rochester, N.Y., which simulates a collection of real forms filled out by respondents, but for which the “truth” is already known, thus rendering unnecessary the laborious and expensive “truthing” of large volumes of actual production data. However, the use of production image snippets for which the correct answers have been determined (i.e., they have been “truthed”) is also possible in the scope of this invention, if desired.
The invention is preferably practiced by incorporation of a Digital Test Deck® such as described in U.S. patent application Ser. No. 10/933,002 for HANDPRINT RECOGNITION TEST DECK, published on Mar. 2, 2006 under the publication number U.S. 2006/0045344 A1 and hereby incorporated by reference.